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The  Defense  Acquisition  University  (DAU)  is  an  integral  compo¬ 
nent  in  the  career  of  every  Defense  Acquisition  Workforce 
member,  from  the  time  they  enroll  in  their  first  DAU  course 
until  they  retire.  One  of  the  many  keys  to  DAU’s  success  is  its 
ability  to  measure  the  effectiveness  of  its  training  programs, 
monitor  performance,  and  improve  its  curriculum.  To  this  end, 
the  authors  conducted  a  data  mining  exercise  within  the  training 
evaluation  data  to  determine  the  key  drivers  of  its  success.  This 
article  explains  the  methodological  approach  used  (structural 
equation  modeling)  as  well  as  the  results,  recommended 
actions,  and  outcomes.  Within  the  DAU  learning  enterprise, 
more  than  326,000  training  events  were  evaluated  during  19 
months  between  January  1,  2008,  and  July  30,  2009.  Results 
indicate  that  DAU’s  learning  enterprise  positively  influences 
job  impact  and  business  results. 
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The  Defense  Acquisition  University  (DAU)  is  critical  to  ensur¬ 
ing  the  Defense  Acquisition  Workforce  is  well  trained  to  meet  our 
nation’s  needs.  As  such,  DAU  is  fully  integrated  in  the  careers  of  its 
workforce  from  the  time  they  enroll  in  their  first  DAU  course  until 
they  retire.  One  of  the  many  keys  to  DAU’s  success  is  its  ability  to 
measure  the  effectiveness  of  its  training  programs,  monitor  perfor¬ 
mance,  and  improve  its  curriculum  (DAU,  2010).  To  this  end,  DAU 
conducted  a  data  mining  exercise  with  its  training  evaluation  data 
to  determine  the  key  drivers  of  its  success. 

DAU  measures  and  monitors  its  own  performance  by  administer¬ 
ing  a  state-of-the-art,  end-of  course  survey  instrument,  which  is  a 
Web-based  learning  evaluation  system  with  an  extensive  database 
of  performance  benchmarks  collected  from  student  survey  data. 
DAU  evaluates  customer  satisfaction  based  on  the  4-level  Kirkpat¬ 
rick  training  assessment  model  and  uses  survey  items  on  a  7-point 
Likert-type  scale  (Kirkpatrick,  1998).  Students  are  provided  a  link  to 
the  survey  at  the  end  of  each  course,  which  includes  questions  on 
course  content,  quality  of  faculty,  and  job  applicability.  Ratings  are 
reviewed  regularly,  and  improvements  are  made  in  DAU’s  learning 
products  and  services  based  on  these  evaluations. 

This  study  focuses  on  the  evaluation  of  key  drivers  for  success¬ 
ful  training  events.  An  advanced  statistical  approach  called  causal 
modeling  (i.e.,  structural  equation  modeling)  was  used  to  determine 
relationships  among  latent  constructs  and  isolate  likely  paths  of 
causation.  The  main  objectives  of  this  research  study  were  fourfold. 

1.  Evaluate  the  survey  instrument  DAU  uses  and  determine 
whether  or  not  it  is  a  valuable  instrument  to  provide  infor¬ 
mation  for  decision  making; 

2.  Assess  the  relationship  between  job  performance  and  impact 
as  perceived  by  participants  attributed  to  DAU  training; 

3.  Access  the  antecedents  and  outcomes  of  learning  and  pro¬ 
vide  a  benchmarking  analysis  of  DAU  scores  versus  other 
organizations;  and 

4.  Provide  recommendations  for  isolated  points  of  intervention, 
which  will  yield  the  largest  improvements  for  the  learning 
enterprise. 


Development  of  Hypotheses 

Although  the  investment  in  learning  by  various  organizations  is 
far  from  consistent  across  industries  (or  even  across  departments 
within  the  same  organization),  few  would  argue  as  to  its  importance. 
The  resultant  training  and  development  budget  isolates  this  invest¬ 
ment  and  is  often  referred  to  when  senior  leaders  are  questioned 
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as  to  their  commitment  for  human  capital  development.  The  two 
primary  expenditures  related  to  training  are  instruction  and  materi¬ 
als.  High  levels  of  instructor  quality  are  synonymous  with  effective 
learning.  This  positivist  relationship  is  at  the  foundation  of  why 
instructors  are  continually  evaluated.  The  performance  of  a  teacher 
as  expressed  by  a  student  after  a  course  is  complete  is  a  universally 
adopted  founding  tradition  of  every  educational  institution.  For  this 
reason,  a  common  expectation  is  the  existence  of  a  positive  rela¬ 
tionship  between  instructor  effectiveness  and  individual  learning. 

Hypothesis  No.  1 

There  is  a  positive  relationship  between  instructor  effectiveness 
and  individual  learning.  In  addition  to  the  quality  of  instruction, 
courseware  quality  is  also  an  important  antecedent  to  learning. 
Whether  the  materials  are  physical  in  the  form  of  books  and  notes, 
or  online,  students  grasp  difficult  concepts  by  reading  them  over 
and  over  again.  While  the  instructor  may  reinforce  the  importance 
of  the  text,  the  explicit  documents  act  as  a  permanent  record  of  the 
content  that  a  student  is  expected  to  master.  As  such,  four  addi¬ 
tional  important  hypotheses  regarding  business  results  and  impact 
were  also  tested. 

Hypothesis  No.  2 

There  is  a  positive  relationship  between  courseware  quality 
and  individual  learning.  Given  the  assumption  that  an  instructor  is 
competent  and  that  course  materials  are  adequate,  students  often 
have  pre-conceived  notions  with  regards  to  the  value  of  a  course 
before  it  has  been  completed.  To  be  accurate,  the  perception  of  a 
worthwhile  investment  is  more  than  just  the  cost  of  the  registration 
fee.  In  most  cases,  the  opportunity  cost  of  time  while  sitting  through 
a  course  (and  therefore,  not  doing  the  job)  is  often  more  valuable 
to  the  learner.  Only  when  both  these  perceptions  (i.e.,  the  cost  of 
registration  and  the  opportunity  cost  of  time)  are  deemed  to  be  fair 
and  adequate,  can  a  student  realize  a  satisfactory  learning  experi¬ 
ence.  Therefore,  the  perception  of  a  worthwhile  investment  is  also 
expected  to  have  a  positive  relationship  with  individual  learning. 

Hypothesis  No.  3 

There  is  a  positive  relationship  between  worthwhile  investment 
and  individual  learning.  Sustainable  high  levels  of  organizational 
performance  can  be  attributable  in  large  part  to  a  superior  learning 
enterprise  that  transforms  human  capital  development  into  action¬ 
able  job  impact  and  business  results  (Bontis  &  Fitz-enz,  2002).  It 
follows  then  that  an  expected  positive  relationship  between  individ¬ 
ual  learning  and  job  impact  and  business  results  should  be  realized. 
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The  issue  here  is  one  of  temporal  lag.  How  do  learners  know  for 
sure  if  a  course  will  impact  their  job  later?  One  way  to  deal  with  this 
limitation  is  to  assess  the  outcomes  longitudinally.  In  other  words, 
provide  learners  with  an  opportunity  to  predict  an  outcome  imme¬ 
diately  after  the  course  is  complete,  and  then  again  sometime  into 
the  future,  a  retrospective  analysis.  As  such,  the  following  important 
hypotheses  regarding  business  results  and  impact  were  also  tested: 

Hypothesis  No.  4 

There  is  a  positive  relationship  between  individual  learning  and 
future  job  impact. 

Hypothesis  No.  5 

There  is  a  positive  relationship  between  individual  learning  and 
future  business  results. 

Hypothesis  No.  6 

There  is  a  positive  relationship  between  individual  learning  and 
actual  job  impact. 

Hypothesis  No.  7 

There  is  a  positive  relationship  between  individual  learning  and 
actual  business  results. 


Method 

When  analyzing  large  sets  of  data,  a  variety  of  statistical  tech¬ 
niques  are  available.  One  common  approach  used  by  researchers 
is  null  hypothesis  testing  with  experimental  and  quasi-experimental 
designs  (Shadish,  Cook,  &  Campbell,  2002).  An  alternative  approach 
is  data  modeling  (Rodgers,  2010).  Structural  equation  modeling 
(SEM)  is  one  such  approach  and  is  useful  for  many  reasons.  Fore¬ 
most  is  its  ability  to  test  multiple  hypotheses  simultaneously  and 
produce  a  visual  model  of  the  causal  relationships  within  a  data  set. 
While  the  benefits  of  SEM  are  plentiful,  a  significant  level  of  inter¬ 
pretation  of  these  models  is  necessary.  In  its  simplest  form,  SEM  is 
an  advanced  statistical  technique  that  computes  the  mathematical 
relationships  among  multiple  variables  simultaneously  in  order  to 
describe  a  chain  of  causation. 

The  measurement  and  structural  models  were  estimated  by 
using  Partial  Least  Squares  (PLS).  PLS  is  a  second  generation  SEM 
technique  that  has  received  positive  recognition  in  the  scientific 
community  (Chin,  1998;  Gefen,  Straub,  &  Boudreau,  2000).  PLS 
was  developed  by  Wold  (1975),  and  it  focuses  on  maximizing  the 
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variance  of  the  dependent  variables  explained  by  the  independent 
ones.  It  is  a  rigorous  SEM  technique  that  requires  only  minimal 
assumptions  about  the  distribution  of  the  data.  PLS  has  five  main 
advantages  over  other  covariance  methods  (e.g.,  LISREL,1  AMOS,2 
etc.)  for  this  research  study:  (a)  it  does  not  assume  normally  dis¬ 
tributed  raw  data;  (b)  the  presence  of  multicollinearity  in  the  data 
is  handled  well;  (c)  it  is  better  suited  to  explain  complex  exploratory 
relationships;  (d)  it  allows  variable  weights  to  scale  for  indicators; 
and  (e)  it  allows  the  use  of  noninterval  scales.  The  raw  data  set  that 
was  to  be  analyzed  fit  well  with  the  corresponding  advantages  of 
PLS.  Furthermore,  PLS  is  often  used  in  exploratory  research  with 
the  ultimate  goal  to  maximize  the  explanatory  power  of  the  resul¬ 
tant  model.  PLS  also  benefits  from  considering  all  path  coefficients 
simultaneously,  allowing  analysis  of  direct,  indirect,  and  spurious 
relationships  and  the  estimation  of  multiple  individual  item  load¬ 
ings  in  the  context  of  a  theoretically  specified  model  rather  than 
in  isolation  (enabling  researchers  to  avoid  biased  and  inconsistent 
parameter  estimates  for  equations). 


Results 

Whereas  in  traditional  path  analysis  the  calculation  of  reliability 
and  validity  statistics  is  independent  of  the  model  being  tested,  PLS 
generates  a  variety  of  reliability  and  validity  statistics  calculated  in 
the  context  of  the  theoretical  model  under  investigation.  To  vali¬ 
date  the  measurement  model,  the  authors  executed  the  following 
series  of  steps.  First,  construct  reliability  was  assured  by  calculating 
Cronbach’s  alpha  values  for  each  construct.  Cronbach’s  alpha  is 
often  used  to  confirm  that  respondents  interpreted  the  meaning  of 
survey  items  accurately,  and  would  continue  to  do  so  in  the  future. 
In  other  words,  survey  items  were  understood  by  the  respondents 
and  tended  to  hang  together  in  a  cluster.  A  Cronbach’s  alpha  value 
exceeding  0.70  is  considered  the  minimum  threshold  (Cronbach, 
1951).  Table  la  outlines  that  all  items  and  their  corresponding  con¬ 
structs  used  in  this  study  exceeded  0.70  (in  fact,  most  exceeded 
0.90). 

Loading  values  (lamdas)  were  used  to  measure  the  validity  of 
items.  This  test  examines  whether  or  not  the  survey  items  used  to 
evaluate  the  effectiveness  of  the  instructor  (four  items  in  this  case) 
load  on  to  an  overall  construct  about  the  instructor.  The  opposite 
would  be  if  a  survey  item  used  to  measure  the  quality  of  instructors 
actually  did  a  better  job  of  measuring  courseware  quality.  Again,  a 
measure  of  0.70  or  higher  is  desired,  and  this  minimum  threshold 
was  exceeded  in  all  cases  (Nunnally,  1978).  The  results  of  Table  1b 
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TABLE  1A.  MEASURES  OF  RELIABILITY  AND  VALIDITY  FOR 
EXOGENOUS  CONSTRUCTS 


ID 

Metric  (Base,  DAU) 

Lamda 

Instructor  Effectiveness  (Alpha  =  0.946,  0.931) 

Base 

DAU 

1058P 

The  instructor  was  knowledgeable  about 
the  subject. 

0.895 

0.910 

1059P 

The  instructor  was  prepared  and  organized 

for  the  class. 

0.885 

0.923 

1269P 

The  instructor  was  responsive  to 
participants'  needs  and  questions. 

0.856 

0.913 

1270P 

The  instructor's  energy  and  enthusiasm 
kept  the  participants  actively  engaged. 

0.851 

0.906 

Courseware  Quality  (Alpha  =  0.899,  0.798) 

1065P 

The  examples  presented  helped  me 

understand  the  content. 

0.886 

0.805 

2726P 

The  scope  of  the  material  was  appropriate 
to  my  needs. 

0.891 

0.806 

2730P 

The  participant  materials  (manual, 
handouts,  etc.)  will  be  useful  on  the  job. 

0.845 

0.800 

2924P 

The  material  was  organized  logically. 

0.892 

0.779 

Worthwhile  Investment  (Alpha  =  0.958,  0.971) 

2743P 

This  training  was  a  worthwhile  investment 
in  my  career  development. 

0.980 

0.986 

2744P 

This  training  was  a  worthwhile  investment 
for  my  employer. 

0.979 

0.986 

Individual  Learning  (Alpha  =  1.000, 1.000) 

919P 

1  learned  new  knowledge  and  skills  from 
this  training. 

1.000 

1.000 

illustrate  that  both  the  survey  instrument  and  DAU  models  used 
valid  and  reliable  measurement  instruments.  In  essence,  the  psy¬ 
chometric  evaluation  of  the  scales  used  in  this  study  was  successful 
and  therefore  adequate  for  model  interpretation. 

Survey  Instrument  vs.  DAU  Model  Interpretation 

In  2009,  Dr.  Nick  Bontis  created  a  predictive  learning  analytics 
model  that  describes  the  relationship  between  training  and  business 
performance  (Bontis  &  KnowledgeAdvisors,  2009).  The  analysis 
proved  successful  and  the  resulting  model  is  depicted  in  Figure  1.  As 
one  might  expect,  the  model  is  complex.  However,  the  model  is  rela¬ 
tively  easy  to  decipher  with  one  key  piece  of  information.  The  chain 
of  causation  lies  along  the  pathways  with  the  highest  beta  values. 
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TABLE  IB.  MEASURES  OF  RELIABILITY  AND  VALIDITY  FOR 
ENDOGENOUS  CONSTRUCTS 


ID 

Metric 

Lamda 

Perceived  Future  Job  Impact  (Alpha  =  0.832,  0.833) 

Base 

DAU 

712P 

1  will  be  able  to  apply  the  knowledge  and  skills 
learned  in  this  class  to  my  job. 

0.726 

0.749 

1279P 

How  critical  is  applying  the  content  of  this 
training  to  your  job  success?  0%-100% 

0.786 

0.925 

1423P 

What  percent  of  your  total  work  time  requires 
the  knowledge  or  skills  presented  in  this 
training?  0%-100% 

0.840 

0.898 

2788P 

What  percent  of  new  knowledge  and  skills 
learned  from  this  training  do  you  estimate  you 
will  directly  apply  to  your  job?  0%-100% 

0.873 

0.907 

Perceived  Future  Business  Results  (Alpha  =  0.775,  0.805) 

2740  P 

Estimate  how  much  you  expect  your  job 
performance  related  to  the  course  subject 
matter  to  improve  in  the  next  12  months. 

0%-100% 

0.909 

0.928 

2741 P 

Based  on  your  response  to  the  prior  question, 
how  much  of  the  improvement  will  be  a  direct 
result  of  this  training,  as  opposed  to  other 

factors?  0%-100% 

0.898 

0.901 

Job  Impact  in  60  days  (Alpha  =  0.892,  0.929) 

1737F 

What  percent  of  your  total  work  time  have  you 
spent  on  tasks  that  require  the  knowledge/ 
skills  presented  in  the  training?  Check  only  one. 

0%-100% 

0.889 

0.932 

1738F 

On  a  scale  of  0%  (not  at  all)  to  100%  (extremely 
critical),  how  critical  was  applying  the  content  of 
the  training  to  your  job  success?  Check  only  one. 

0.916 

0.934 

2818F 

What  percent  of  new  knowledge  and  skills 
learned  from  this  training  did  you  directly  apply 
to  your  job?  Check  only  one.  0%-100% 

0.912 

0.942 

Business  Results  in  60  days  (Alpha  =  0.708,  0.814) 

2751 F 

Given  all  factors,  including  this  training,  estimate 
how  much  your  job  performance  related  to  the 
course  subject  matter  has  improved  since  the 
training.  0%-100% 

0.906 

0.931 

2502F 

Based  on  your  response  to  the  prior  question, 
estimate  how  much  of  the  improvement  was  a 
direct  result  of  this  training.  0%-100% 

0.845 

0.905 
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Note.  Adapted  from  The  Predictive  Learning  impact  Model  by  N.  Bontis  and 
KnowledgeAdvisors,  Inc.  Copyright  2009  by  KnowledgeAdvisors,  Inc. 


Before  examining  the  model,  a  description  of  the  data  source  is 
essential.  Data  used  in  the  analysis  were  extracted  from  the  survey 
instrument  system.  Using  Web-hosted  surveys,  learners  provided 
feedback  about  their  training  experience  immediately  after  training 
and  60  days  after  training.  The  immediate  survey  asked  learners 
to  rate  the  quality  of  their  training  experience  as  well  as  predict 
whether  training  would  improve  their  job  performance  and,  in  turn, 
contribute  to  business  results.  In  the  model  shown  in  Figure  1,  the 
immediate  survey  results  contribute  to  the  following  factors  about 
training:  Instructor  Effectiveness,  Courseware  Quality,  Worthwhile 
Investment,  and  Individual  Learning. 

The  immediate  responses  also  contributed  estimates  of  future 
job  performance  and  estimated  impacts  on  business  results:  Per¬ 
ceived  Future  Job  Impact  and  Perceived  Future  Business  Results. 
The  far  right  side  of  the  model  has  two  factors,  Job  Impact  in  60 
Days  and  Business  Results  in  60  Days,  which  represent  retrospective 
input  gathered  from  learners  after  they  have  had  60  days  to  apply 
their  learning  on  the  job.  More  than  a  million  data  points  were  used 
in  the  survey  instrument  base  model  with  learners  assessed  from 
many  well-known,  globally  recognized  companies  (e.g.,  Microsoft, 
HSBC,  Caterpillar,  and  BAE). 
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The  left  side  of  the  model  represents  the  three  most  important 
antecedent  aspects  of  training:  Worthwhile  Investment,  Courseware 
Quality,  and  the  learner’s  perspective  about  Instructor  Effectiveness. 
But  how  do  these  three  factors  contribute  to  individual  learn¬ 
ing?  Effective  instructors  contribute  by  developing  high-quality 
courseware  (B  =  0.556)  (e.g.,  materials,  delivery  format,  learning 
environment,  etc.).  Both  contribute  to  the  learner’s  perception  that 
training  was  a  worthwhile  investment  (B  =  0.625).  While  each  factor 
also  has  a  direct  relationship  with  individual  learning,  the  strongest 
relationship  is  through  the  last  factor— worthwhile  investment  (B  = 
0.483). 

The  second  half  (right  side)  of  the  model  represents  how 
individual  learning  influences  job  performance  and  business  (orga¬ 
nizational)  outcomes.  By  examining  the  values,  we  see  from  the 
nondominant  (lowest  values)  paths  that  individual  learning  does 
not  have  a  strong  direct  effect  on  Job  Impact  and  Business  Results 
directly  (both  far  right)  60  days  after  training.  However,  by  follow¬ 
ing  the  dominant  paths,  we  see  that  Individual  Learning  leads  to 
Perceived  Future  Job  Impact,  actual  Job  Impact  in  60  Days  and 
Business  Results  in  60  Days.  This  path  indicates  that  knowledge 
for  the  sake  of  knowledge  (Newman,  1947)  is  not  sufficient.  Train¬ 
ing  must  be  perceived  as  relevant,  practical,  and  applicable  before 
learners  will  indicate  it  will  (and  does)  have  an  impact  on  perfor¬ 
mance  and  eventually  business  results.  The  entire  causal  pathway 
for  the  model  looks  like  a  giant  N,  starting  at  the  bottom  left,  rising 
to  the  top  left,  slanting  diagonally  to  the  bottom  right,  and  then  up 
to  the  top  right. 

Does  the  model  effectively  describe  the  relationship  among 
variables?  Yes,  the  model  has  a  relatively  high  explanatory  power 
for  predicting  Job  Impact  in  60  days  (R2  =  40.0%)  and  Business 
Results  in  60  days  (R2  =  59.2%).  A  model  that  predicts  the  structure 
and  relationships  perfectly  would  have  an  R2  value  of  100  percent, 
although  this  situation  is  virtually  impossible  to  achieve  in  social 
science  research.  To  put  this  measure  in  perspective,  consider  that 
it  is  an  algorithm  you  could  use  at  a  casino.  Given  the  value  of  the 
cards  in  your  hand  (e.g.,  Instructor  Effectiveness,  Courseware  Qual¬ 
ity,  etc.),  you  would  win  59.2  percent  of  the  time. 

This  predictive  learning  analytics  model  (KnowledgeAdvisors, 
Inc.)  is  relevant  to  DAU  because  it  serves  as  a  base  model  (derived 
from  a  data  set  of  over  1  million  surveys  industry-wide)  for  bench¬ 
marking  purposes.  As  a  result,  the  DAU  can  compare  its  model  and 
results  to  the  KnowledgeAdvisors’  predictive  learning  analytics 
model  to  determine  where  it  can  improve  and  where  it  is  outper¬ 
forming  other  organizations. 
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The  DAU  Model 

Each  year  the  DAU  collects  hundreds  of  thousands  of  evalu¬ 
ations  after  training  events  to  determine  whether  the  curriculum 
and  its  outcomes  were  effective.  Immediate  postcourse  evalua¬ 
tions  are  deployed  as  well  as  60-day  follow  up  evaluations.  In  this 
way,  learners  provide  feedback  about  the  quality  of  the  course  and 
predict  whether  they  will  apply  what  they  learned.  On  the  60-day 
follow  up  evaluation,  learners  indicate  whether  training  contributed 
to  improved  job  performance  and  business  results.  For  this  study 
more  than  326,000  evaluations  were  collected  during  19  months 
between  January  1,  2008,  and  July  30,  2009. 

Figure  2  shows  the  DAU  model  with  KnowledgeAdvisors  bench¬ 
mark  values  above  the  DAU  model  values.  The  analysis  reveals  many 
important  facts  about  the  DAU’s  curriculum. 

FIGURE  2.  PREDICTIVE  LEARNING  ANALYTICS  MODEL-DAU 

AND  SURVEY  INSTRUMENT  MODEL  BENCHMARK 


First,  the  causal  chain  depicted  in  the  model  explains  the  rela¬ 
tionships  among  the  data  well.  In  fact,  the  model  fits  the  DAU’s 
data  better  than  the  survey  instrument  benchmark  data.  (This  is  not 
unusual;  the  model  will  fit  some  data  sets  better  or  worse  than  the 
benchmark.)  The  model  predicts  52.7  percent  of  the  Job  Impact  in 
60  Days  and  67.9  percent  of  the  Business  Results  in  60  Days.  The 
values  for  the  KnowledgeAdvisors  Benchmark  data  are  lower  at 
40.0  percent  and  59.2  percent,  respectively.  These  results  indicate 
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that  the  key  aspects  of  training  that  drive  job  impact  and  business 
performance  hold  true  for  both  the  KnowledgeAdvisors  Benchmark 
data  and  the  DAU  data.  As  a  refresher,  Instructor  Effectiveness  links 
to  Courseware  Quality,  which  links  to  Worthwhile  Investment.  All 
three  in  that  order  optimize  Individual  Learning.  In  turn  Individual 
Learning  leads  to  Perceived  Future  Job  Impact,  Job  Impact  in  60 
Days,  and  then  Business  Results  in  60  Days  in  that  order.  To  improve 
the  effectiveness  of  courses— at  least  in  terms  of  increasing  job 
performance  and  business  impact— DAU  should  focus  on  Instructor 
Effectiveness,  Courseware  Quality,  and  ensure  that  learners  perceive 
that  training  is  a  Worthwhile  Investment.  Improvement  actions  will 
be  discussed  later  in  this  article. 

Second,  the  strength  of  the  causal  relationships  is  somewhat 
stronger  for  the  DAU  compared  to  the  KnowledgeAdvisors  Bench¬ 
mark  for  five  relationships  (arrows).  These  are  designated  with  a 
superscripted  a  after  the  value.  For  only  one  relationship,  the  link 
between  Courseware  Quality  and  Individual  Learning,  the  DAU  value 
is  lower  than  the  survey  instrument  benchmark  and  is  indicated  by  a 
superscripted  b.  When  this  relationship  was  examined  in  more  detail, 
it  was  discovered  that  Courseware  Quality  was  more  important  for 
younger  learners  and  less  so  for  older  learners.  Younger  learners 
preferred  e-learning,  whereas  older  learners  preferred  traditional 
classrooms  and  effective  instructors.  Interestingly,  younger  learn¬ 
ers  indicated  that  training  had  a  greater  impact  on  job  impact  and 
business  impact  60  days  after  training. 

Third,  DAU  instructors  have  a  strong  influence  on  Individual 
Learning  and  eventually  Job  Impact  and  Business  Results.  In  fact, 
when  compared  to  the  KnowledgeAdvisors  benchmark  (0.083), 
the  relationship  between  Instructor  Effectiveness  and  Individual 
Learning  (B  =  0.163)  is  almost  twice  as  large  for  DAU.  A  stronger 
relationship  between  Instructor  Effectiveness  and  Courseware  Qual¬ 
ity  still  exists,  but  by  comparing  the  magnitude  of  the  relationship 
between  the  DAU  and  the  benchmark,  clearly,  instructors  hold  more 
influence  within  DAU  than  at  other  organizations. 

Fourth,  guest  speakers  also  impact  learning.  When  guest  speak¬ 
ers  taught  courses,  higher  levels  of  Individual  Learning  occurred. 
When  guest  speakers  were  not  included,  Job  Impact  and  Business 
Results  were  generally  lower  than  the  survey  instrument  benchmark. 

Fifth,  application  is  a  critical  element  to  successful  courses.  H ig h 
job  application  scores  were  linked  to  high  learning  scores,  extremely 
high  Job  Impact  scores,  and  Business  Results  scores. 

Sixth,  application  is  also  strongly  linked  to  whether  learners  rec¬ 
ommend  courses  for  future  learners.  When  recommendation  scores 
were  low,  the  Business  Results  in  60  Days  were  also  lower.  This  is 
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a  strong  indicator  that  the  course  is  not  meeting  individual  needs 
and  organizational  needs  and  therefore  should  be  revised  or  retired. 

Seventh,  Defense  Acquisition  Workforce  Improvement  Act 
(DAWIA)  levels  were  investigated  to  test  for  their  influence  on 
training  outcomes.  DAWIA  specifies  three  skill  levels  (Basic,  Inter¬ 
mediate,  and  Advanced)  associated  with  13  career  fields  in  the 
acquisition  system.  No  consistent  pattern  of  influence  emerged 
across  the  model  for  the  three  DAWIA  levels. 

Eighth,  the  educational  level  of  learners  (e.g.,  high  school,  col¬ 
lege,  or  graduate  school)  does  influence  outcomes  of  the  model. 
Learners  with  some  graduate  education  are  more  critical  of  instruc¬ 
tors  and  appreciate  good  courseware.  Learners  with  a  high  school 
education  have  the  lowest  perception  that  training  is  a  Worthwhile 
Investment ,  but  yield  the  highest  response  to  Individual  Learning. 
Interestingly,  this  group  scored  much  lower  than  the  benchmark 
regarding  future  results. 

Lastly,  the  DAU’s  course  offerings  and  their  influence  on 
job  performance  and  outcomes  were  evaluated  for  longitudinal 
improvement.  Indeed,  it  was  confirmed  that  scores  improved  from 
2008  to  2009.  In  fact,  scores  for  every  category  improved  except 
for  Instructor  Effectiveness,  which  was  already  high. 


Recommended  Actions 

An  important  and  useful  finding  of  this  study  indicates  that 
the  key  aspects  of  training  drive  Job  Impact  and  Business  Results. 
This  in  itself  is  valuable,  but  such  value  quickly  fades  if  insights 
cannot  be  turned  into  action  to  improve  the  curriculum.  Table  2 
provides  a  summary  of  the  results  of  this  study  as  well  as  recom¬ 
mended  actions.  If  the  DAU  pursues  these  actions,  the  curriculum  is 
likely  to  improve  as  evidenced  by  improved  scores  on  the  training 
evaluations. 


Conclusions 

In  its  evolution,  DAU  has  broadly  embraced  adult  learning 
designs  in  its  formal  courses  and  accepted  the  fact  that  adults  learn 
best  “by  doing,”  whether  in  the  formal  learning  environment  or  on 
the  job.  With  formal  training,  DAU  attempts  to  “train  as  the  work¬ 
force  should  work,”  and  prepares  the  workforce  to  “work  as  they 
are  trained”  by  using  the  same  training  tools  and  learning  assets  at 
their  individual  places  of  work  that  they  formerly  used  in  the  class¬ 
room.  This  study  provides  strong  evidence  that  the  key  aspects  of 
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TABLE  2.  RECOMMENDED  ACTIONS 


Results 

Recommended  Action 

Application  is  a 

critical  element  of 

training 

As  appropriate,  DAU  can  improve  its 
impact  on  job  performance  and  business 
results  by  increasing  the  opportunities  to 
apply  what  is  learned  during  training. 

Courseware  Quality 
is  more  important  for 
younger  learners 

To  improve  learning  among  younger 
learners,  invest  in  self-study  modules  and 
quality  courseware. 

DAU  instructors  have 

a  strong  influence  on 

older  learners 

For  instructor-led  courses,  especially  with 
older  learners  as  the  target  audience, 
invest  time  and  effort  to  find  high-quality 
instructors  who  can  effectively  teach  the 
materials  regardless  of  the  quality  of  the 

courseware. 

Guest  speakers  also 
impact  learning 

When  appropriate,  use  guest  speakers  to 
augment  or  lead  instructor-led  courses 
for  older  learners.  Guest  speakers  tend 
to  have  more  impact  on  learning  than  the 

standard  cadre  of  instructors. 

Learners  recommend 

effective  training 

Use  the  question,  “1  would  recommend 
this  training  to  other  learners”  as  a  leading 
indicator  of  the  quality  of  training  and 
whether  it  will  lead  to  job  performance 
and  business  impact.  If  the  rating  for  this 
question  is  low  for  a  given  course,  it  should 
be  revised  with  a  focus  on  improving  the 
ability  to  apply  what  is  learned  during 
training. 

DAWIA  levels  do  not 

influence  training 
impacts 

When  building  DAU  courses,  it  is  not 
necessary  to  consider  the  DAWIA  level  of 
the  audience.  Other  factors  like  age  and 

education  are  more  influential  than  DAWIA 

levels. 

Education  impacts 
learning  and 
performance 

To  ensure  that  training  leads  to 
performance  and  future  results,  courses 

should  be  tailored  to  the  educational  level 

of  the  audience. 

361 


A  Publication  of  the  Defense  Acquisition  University 


http://www.dau.mil 


DAU’s  approach  to  training  drive  Job  Impact  and  Business  Results. 
Having  empirical  evidence  derived  from  relatively  large  data  sets  is 
very  useful  in  rationalizing  the  cost  of  training  regarding  improving 
performance  on  the  job.  Additionally,  confirming  the  reliability  and 
validity  of  the  survey  instrument  is  important  to  any  DAU  curricula 
and  recourse  decisions  based  on  survey  instrument  scores  as  well 
as  other  considerations. 

Given  that  Job  Impact  results  were  based  on  self-reporting 
perceptions  and  not  an  independent  external  measure  (not  within 
the  instrument),  the  outcome  is  still  very  strong  in  its  implications, 
largely  due  to  the  size  of  the  sample  as  well  as  previous  relation¬ 
ship  studies  concerning  the  close  relationships  between  measured 
perception  and  actual  reality.  Dess  and  Robinson  (1984)  indicate 
that  such  perceived  measures  of  business  results  are  reasonable 
surrogates  for  more  tangible  and  objective  measures  of  business 
outcomes  (e.g.,  revenue  growth,  profits).  Others  (Geringer  &  Hebert, 
1989;  Hansen  &  Wernerfelt,  1989;  Lyles  &  Salk,  1997;  Venkatra- 
man  &  Ramanujam,  1987)  have  demonstrated  that  such  perceived 
measures  also  are  positively  correlated  with  objective  financial 
performance  metrics. 

Finally,  the  recommendations  discussed  previously  are  now 
being  incorporated  within  DAU  course  development  and  course 
update  design  strategies.  Therein  lies  the  power  of:  (a)  bench¬ 
marking  learning  data  across  a  very  large  set  of  comparative  peer 
organizations;  and  (b)  using  structural  equation  modeling  to  ascer¬ 
tain  specific  points  of  intervention  for  evaluating  and  improving  the 
learning  enterprise,  thereby  assuring  a  healthy  return  on  investment 
for  training  dollars. 
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ENDNOTES 

1.  LISREL,  an  acronym  for  linear  structural  relations,  is  a  statistical  software  package  used 
in  structural  equation  modeling.  LISREL  was  developed  in  the  1970s  by  Karl  Joreskog, 
then  a  scientist  at  Educational  Testing  Service  in  Princeton,  NJ,  and  Dag  Sorbom,  later 
both  professors  of  Uppsala  University,  Sweden. 

2.  AMOS,  an  acronym  for  analysis  of  moment  structures,  is  designed  primarily  for  structural 
equation  modeling,  path  analysis,  and  covariance  structure  modeling,  though  it  may  be 
used  to  perform  linear  regression  analysis.  It  features  an  intuitive  graphical  interface  that 
allows  the  analyst  to  specify  models  by  drawing  them.  It  also  has  a  built-in  bootstrapping 
routine  and  superior  handling  of  missing  data.  It  reads  data  from  a  number  of  sources, 
including  MS  Excel®  spreadsheets. 
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